A depth map is a digital representation of the distances between the positions of various objects of a scene and a receiver, the rendition of which is comparable to a photograph bearing depth information and not brightness information.
Various techniques exist for acquisition or determination of a depth map, for example, stereoscopy and time of flight measurement.
Time of flight measurement consists in emitting onto a scene an identifiable electromagnetic wave signal, generally pulsed laser illumination, and detecting the signals reflected by the objects of the scene. The time difference between the moment of emission of a signal and the moment of reception of that signal reflected by an object of the scene enables calculation of the distance separating the transmitter-receiver from the object.
For its part, stereoscopy is a so-called passive measurement technique and enables determination of a depth map of a scene from two photographic images of the scene taken from different viewpoints.
FIG. 1 illustrates the principle of determination of depths by stereoscopy. Two images 201, 202 of the same scene 10 including an object 11 are obtained from different known viewpoints, for example, by means of image sensors including optics. The two viewpoints usually have a parallel optical axis and are aligned horizontally to correspond to a left image 201 and a right image 202 separated by a so-called fundamental deviation.
The parallax resulting from the fundamental deviation means that the object 11 is projected into the images 201, 202 at respective different positions 111, 112. The projections 111, 112 are situated on the same epipolar line 12 that is typically horizontal for viewpoints situated in the same plane with vertically aligned receiving surfaces and parallel optical axes.
The determination of the distance between the receiver and the object 11, i.e., the depth of the object 11, therefore includes for each point of the image 201 a calculation of similarity with the points of the image 202 situated on the same epipolar line 12. The distance δd between the positions of the similar points of the two images 201, 202 is referred to as the disparity. The value of the disparity δd between the two projections 111, 112 enables extrapolation of the depth of the corresponding object 11, notably taking into account the fundamental deviation and optical and technological characteristics of the acquisition device.
The techniques for identification of similarities between the projections and of extrapolation of depths from disparities generally include an undersampled initialization calculation, a less sampled calculation exploiting the initialization and refining processing in order to obtain a correspondence by a so-called decreasing granularity method. These techniques are notably described in more detail in the scientific paper: HIRSCHMULLER, Heiko, Stereo processing by semi global matching and mutual information, IEEE Transactions on pattern analysis and machine intelligence, 2008, vol. 30, no 2, p. 328-341.
The calculations employed in the usual forms of stereoscopic processing therefore require a large amount of calculation resource leading to long execution times and sometimes to errors.
Merging the time of flight measurement and stereoscopy technologies can make it possible to limit and to refine the usual forms of stereoscopic processing.
For example, it is possible to produce an a posteriori estimate of the depth map using both the time of flight measurement and the stereoscopic image or to initialize stereoscopic processing with depth information obtained by time of flight measurement using one or more time of flight (ToF) sensors, for example, in place of undersampled processing. Moreover, it is also possible to calculate each disparity value taking into account depth variations measured by time of flight.
This type of combination of technologies is notably described in more detail in the scientific paper: Vineet Gandhi, Jan Cech, Radu Horaud. High-Resolution Depth Maps Based on TOF-Stereo Fusion, ICRA 2012—IEEE International Conference on Robotics and Automation, May 2012, Saint-Paul, Minn., United States. IEEE, pp. 4742-4749, 2012.
It should be remembered that a time of flight sensor enables measurement of the distance between an object and the sensor by measuring the time difference between the emission of a signal and the reception of that signal after it has been reflected by the object.
Current combinations of stereoscopic determination and time of flight measurement techniques necessitate extremely good mutual calibration and pixel to pixel matching between the stereoscopic images and the time of flight measurement. In other words, a high-resolution time of flight sensor, for example, one using a minimum of 25 kilopixels, is required to effect these depth map determination improvements.
Now, high-resolution time of flight sensors are bulky and greedy in terms of energy and calculation resources, which is generally undesirable. They are also costly. Finally, these inconveniences are notably incompatible with the compact and battery-powered autonomous on-board technologies that are increasingly commonly used.
There nevertheless exist time of flight sensors of lower resolution compatible with the requirements of autonomous on-board technology. Moreover, they are less costly than the high-resolution competition.
This type of time of flight sensor is very compact, autonomous and economic in energy (notably consuming less than 20 μW on standby and less than 35 mW in operation). This type of sensor measures a distance map in which the depths are obtained acquisition zone by acquisition zone, for example, in accordance with a matrix of 5*3 zones or 15*9 zones of a scene equivalent image. A distribution of the depths of the objects present in the corresponding part of the scene is generated in the form of a histogram for each acquisition zone.
Accordingly, compared to high-resolution time of flight sensor technologies, the information obtained by this type of sensor is less precise but can be integrated into a processing subsystem of an autonomous or on-board system.
There exists a requirement to improve stereoscopic processing with the assistance of such time of flight measurements obtained with a sensor compatible with an autonomous or on-board technology in a simple and efficient manner.